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enterprise. The TOUS did not perform well in assessing this 
population. Although the posttest reliability, measured by KR-20 was 
a respectable 0.66, the mean difficulty was a high 69.30 and the mean 
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Abstract 



This study, a companion to a qualitative investigation by McArthur (1993), 
examines the role of a six or eight week summer university laboratory internship for 
high ability secondary students in developing the views of these students with respect 
to the nature of science. Students were assessed using a gender-neutral version of the 
Test of Understanding Science (TOUS)(Form W) as part of a pretest-posttest research 
design. The TOUS provides a total score and three subscales (understanding about the 
scientific enterprise, about scientists and about the methods and aims of science). 
Although the TOUS is not directly aligned with instruction it was applied as a measure 
of the degree to which students are able to transfer aspects of their laboratory 
internship to an understanding of science in general. 

The overall TOUS scores shov/ed no statistically significant change, from pretest 
to pusttest, nor were any significant changes seen with respect to any of the subscales. 
When all other variables were analyzed (e.g. ethnicity, number of science classes 
completed, gender, length of internship or academic average, and internship duration) 
only the internship duration (six week versus eight week) showed a significant change. 
Multiple regression and covariance techniques demonstrated that the eight week group, 
although statistically identical with the six week group on the overall pretest, did make 
a significant gain (p < .05) at the posttest lev ?1 on subscale I (understanding about the 
scientific enterprise). No other significant changes were noted in the subscales. 

The TOUS itself did not perform well in assessing this population. Although the 
posttest reliability, measured by KR-20 was a respectable 0.66, the mean difficulty was 
a high 69.30 and the mean discrimination was only 0.23. These values may be 
explained by the generally high level of pretest achievement and the overall 
homogeneity of the students. The accompanying qualitative study indicates that 
students have gained much with respect to the nature of science by working as 
laboratory interns, but the TOUS is clearly not the best instrument to use in 
substantiating that finding. 
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Purpose of the Study 

The purpose of this study is to examine the extent to which an intensive summer 
laboratory internship communicates aspects of the philosophy of science to a group of 60 
high-achieving secondary school students as measured by the Test of Understanding Science 
(TOUS) and its three scales. 

Significance of the Study 
The nature of science is one of the major areas targeted for inclusion in school 
science curricula by principal current science education reform efforts, Project 2061: Science 
for Ail Americans (AAAS, 1989) and Scope Sequence and Coordination from the National 
Science Teachers Association (NSTA, 1992), in the newest national assessment of 
educational progress (NAEP) and in the proposed national standards for science education. 
Cossman (1989) conducted a detailed review of philosophy of science issues contained in 
the NAEP items and in the Project 2061 objectives. Both NAEP and Project 2061 contain 
objectives suggesting that students should be familiar with scientific observation, methods 
of science, data interpretation, support and refutation of scientific ideas, hypothesis 
generation, the tentative nature of scientific ideas and the character of the community of 
scientists. 

Although not featured as prominently as in Project 2061, the Scope, Sequence and 
Coordination reform effort also suggests that the nature of science is an important 
ingredient in the reform of science instruction. In addition, the new National Standards for 
Science Education will include issues related to the philosophy of science as learning goals. 
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Students should "develop an understanding of the nature of scientific inquiry . . ." "as a 
model of how we know what we know in science" (National Research Council, 1993, p. 55). 

In spite of recommendations to include philosophy of science issues in the 
curriculum, none of the sources cited suggest a mechanism by which such recommendations 
may be put into place. The core issue is how to address and communicate recommended 
aspects of the philosophy of science most effectively. The significance of this study lies in 
its potential to show that a student working as an intern in a research laboratory may 
actually learn about the nature of science while doing science. As the number of science 
internships and related experiential programs increases, either through summer institutes 
for high ability students or through academic year mentorships for students of average 
ability (Waltner, 1992) - it becomes increasingly important to assess exactly what students 
gain from such experiences. This study reported here attempts to shed light on one of the 
anticipated outcomes from such programs - an increase in the level of knowledge regarding 
the nature of science. 

Goals of the Study 

The following are the principal areas of investigation with respect to high- 
ability secondary school students participating in a summer laboratory internship. To: 

1) determine the degree to which high-ability students understand the nature of 
science as measured by the TOUS. 

2) explore indicators of the role that a summer laboratory internship experience 
may play in altering students' knowledge of the philosophy of science. 

3) investigate the role played by other factors (gender, internship duration, 
nature of the internship experience, etc.) in altering students' knowledge of 
the nature of science while participating in a laboratory internship. 
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4) study the TOUS instrument as an effective and reliable measure of the nature 
of science for high-ability students. 

Design and Procedures of the Study 

The Method 

This study is a single group pretest-posttest (Campbell and Stanley, 1966); a pre- 
experimental design providing clues to causation. The criterion measure was provided by 
pretest-posttest administrations of the Test of Understanding Science (TOUS) (Cooley and 
Klopfer, 1961). The TOUS was administered as a pretest to students in each of the two 
groups upon their arrival on campus. The posttest was administered to the students several 
days before the completion of both the six week and eight week internships. In addition 
to the administration of the TOUS, information regarding a number of independent 
variables (academic grades, gender, number of science classes completed, duration of 
internship, etc.) was also collected. 

It should be noted that the TOUS was designed to measure student's knowledge of 
the nature of science generally and is not, for that reason, aligned with the laboratory 
internship experience reported here. Any gains seen in TOUS scores from pretest to 
posttest would most accurately be attributed to transfer effects rather than direct 
instruction-related achievement. 
The Sample 

Sixty high ability secondary school students - of whom 56 completed both the pretest 
and posttest - were involved in the study during the summer of 1992 at a large midwestern 
research university. All students were "high-ability" based on measures including class rank, 
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letters of recommendation, prior achievement, and other factors. Most students entered the 
summer internship program with a strong science background. However, only four students 
had actually experienced a similar laboratory internship experience. 

Twenty students participated as a cohort in the laboratory internship for eight 
consecutive weeks; another group of forty students arrived two weeks after the first group 
began, participating for a total of six consecutive weeks. Both groups of students ended 
their internship experiences at the same time. Table 1 provides a summary of some 
characteristics of the students participating in the laboratory internship. 



Please Insert Table 1 Approximately Here 



While the details of the individual internship experiences varied, each student had 
daily contact with researchers in a university laboratory as they worked on some aspect of 
an on-going project. This contact included frequent interaction with research assistants and 
post-doctoral fellows but less frequent direct work with the principal investigator. Students 
operated primarily as laboratory technicians, but all participated in research meetings, were 
briefed by and reported to the principal investigator, and were required to make a final 
presentation at the conclusion of the internship by explaining the goals and results of their 
research experience. In addition to the general presentation required of all students, 
approximately 15 percent of the participants made preseniations of the results of original 
research projects. 
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The Instrument 

In spite of interest expressed by science education reformers and curriculum 
developers that an understanding Gf the nature of science be a part of students' educational 
experience, this domain has been generally neglected by designers of assessment tools. 
Consequently, the best single measure of student's understanding of the philosophy or 
nature of science remains the Test of Understanding Science (TOUS) developed in the 
early 1960's (Cooley and Klopfer, 1961). 

The instrument is a 60-item multiple choice test providing a total score and subscores 
targeting knowledge of the scientific enterprise (I), the scientist (II), and the methods and 
aims of science (III). The characteristics of each of the subscores is outlined below (Cooley 
and Klopfer, 1963, p. 74). 

Subscore I; Understanding about the scientific enterprise (18 items) 

1. Human element in science 

2. Communication among scientists 

3. Scientific societies 

4. Instruments 

5. Money 

6. International character of science 

7. Interaction of science and society 

Subscore II; Understanding about scientists (18 items) 

1. Generalizations about scientists as people 

2. Institutional pressures on scientists 

3. Abilities needed by scientists 

Subscore III; Understanding about the methods and aims of science (24 items) 

1. Generalities about scientific methods 

2. Tactics and strategy of sciencing 

3. Theories and models 

4. Aims of science 



7 



7 

5. Accumulation and falsification 

6. Controversies in science 

7. Science and technology 

8. Unity and interdependence of the sciences 

Cooley and Klopfer (1961) report an overall reliability (KR-20) of 0.76 v/ith KR-20 
scores for the subscales ranging from 0.52 to 0.58 with a large heterogeneous group of 
secondary students. 

Although the TOUS is generally accepted as the most effective instrument available, 
it has not escaped criticism even at the time of its development. Grobman, writing in Buros 
(1972), states that one of the items designed to find out how much students know about the 
direction of science in America may be problematic since the most appropriate response 
varies depending upon changes in politics and perceived national goals. Wheeler (1968), 
Welch (1969), Noll (in Buros, 1972) and Aikenhead (1973) have made related criticisms of 
the TOUS, but the frequency of its initial use and the paucity of other such instruments 
justifies the use of TOUS in the assessment of nature of science knowledge. 

A reexamination of the TOUS instrument with respect to its use in this study 
revealed potential gender bias in a number of the items. For example, the word scientist 
in the original version (Form W) is frequently linked with the personal pronoun he or his. 
Therefore, the TOUS (Form W) was rewritten slightly to minimize gender bias in those few 
items exhibiting such bias. The gender-neutral version Form W was applied in this study. 
TOUS and Related Studies of the Impact of Investigative Teaching 

During the height of its popularity, the TOUS instrument was applied in a number 
of studies primarily to examine the extent to which one teaching technique or another best 
communicated aspects of the philosophy of science to students. An examination of a 
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number of studies showed that a teaching approach involving an integrated or inquiry / 
investigative component resulted in a significant positive difference in all or most subscores 
of the TOUS when compared with a "traditional" approach. Although it is difficult to 
determine precisely what is meant by a "traditional" approach, many researchers 
characterized it as a didactic, lecture-demonstration mode of teaching. Table 2 contains a 
summary of the results of a number of potentially-related studies in which the TOUS was 
applied. 

Specifically, Sheppard (1969) demonstrated that junior high students taught using the 
Earth Science Curriculum Project (ESCP) - an inquiry approach - fared significantly better 
on all subscores of TOUS when compared with counterparts in "traditional" classes. Green 
(1972) with a similar study, showed that ESCP students performed significantly better than 
did "traditional" students, but only in subareas II (knowledge about scientists) and III 
(knowledge about the methods and aims of science). 



Please Insert Table 2 Approximately Here 



Mallette (1967), examining the junior high school integrated science project, 
Interaction of Matter and Energy (IME), demonstrated that there was a significant gain in 
TOUS scores when comparing the traditional and IME group. The students in the IME 
classes showing growth on the TOUS primarily in Subarea III (knowledge about the 
methods and aims of science). 
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Mallette's work is substantiated by Schlenker (1970) who found that grade 5-8 
students involved in an inquiry-based physical science program achieved significantly better 
on the TOUS - particularly in Subareas I (science) and II (scientists) when compared with 
students in traditional classes. Although these results indicate that an inquiry approach has 
a strong impact on an individual's knowledge of aspects of the nature of science other 
researchers such as Smith (1971), Stengel (1971) and Durkee (1974) examining investigative 
teaching approaches have not found such significant results. Durkee studied a population 
of high-achieving secondary school students in a summer physics internship much like those 
students examined here. 

Results and Analysis 

The results of this study fall into several categories; characteristics of the Test of 
Understanding Science itself for this group of students, an analysis of the TOUS scores, and 
a statistical examination of the contribution of certain independent variables to the overall 
TOUS variance. 
Analysis of the Test Itself 

Table 3 illustrates the reliability, mean difficulty and mean discrimination values for 
this administration of the TOUS instrument. Kuder-Richardson 20 reliability, measures 
indicate considerable variability ranging at the posttest level from 0.38 to 0.57 on the 
subscales, with an overall reliability of 0.66. One might expect the reliability to be higher 
for a professionally-designed test. However, the values established here correspond well 
with those reported by Cooley and Klopfer (1961) for a large (2425 students) heterogenous 
group used n norm the test originally. 
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Difficulty and discrimination measures how well an item distinguishes between those 
students who receive overall high scores and those who receive overall low scores. High 
positive discrimination (near +1.0) means that most of the high-scoring students chose the 
correct answer. The generally-accepted value for a test that discriminates well is +0.40 or 
better (Evaluation and Examination Service, 1992). The mean discriminations value for the 
posttest reported here was 0.23 and ranged from 0.26 to 0.36 on the posttest subscales. 

The item difficulty index is a measure of the percent of students who answer the 
item(s) correctly. Normally, for test likes this, the ideal difficulty is 42.5% (Ebel and 
Frisbie, 1986). Average difficulties on the various posttest scales in the present study varied 
from 60.12 on one of the subscales, up to 76.11%. While both the mean difficulty and mean 
discrimination are outside the normally acceptable ranges for tests designed for general use, 
the computed indexes were expected given the rather homogeneous sample of high ability 
students. 



Please Insert Table 3 Approximately Here 



Student Knowledge of the Nature of Science 

Table 4 summarizes the scores and standard deviations for the total TOUS and for 
the three subscales. Included for comparison are the results frorr : the only other study of 
high-ability summer laboratory interns (Durkee, 1974). In addition, t-tests of significance 
are reported for the overall test and for each of the scales. 
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Please Insert Table 4 Approximately Here 



Although the students in this study are high achievers generally, most secondary 
science curricula contain little about the nature of science and few instructors emphasize 
such issues. There, it was a surprise that students had such high mean scores at the pretest 
level; 68.67% correct overall with a range of 60.12% to 75.79% on the subscales. However, 
these results correspond well with those established for a similar student group by Durkee 
(1974). Furthermore, although the trend is an increase in knowledge regarding the nature 
of science, neither the data reported here nor those of Durkee (1974) illustrate a significant 
change from pretest to posttest. 

The Contributions of Selected Learner-Related Characteristics 

A number of independent variables were evaluated in an attempt to examine their 
role in students' knowledge of the nature of science. Variables such as gender, ethnicity, 
number of high school science classes completed, academic achievement and length of 
internship were analyzed with SPSS / PC+ 4.0 (Norusis, 1990) using two multiple regression 
techniques; stepwise addition and backward deletion. Only when the students are separated 
into groups associated with length of internship (6 weeks or 8 weeks in duration) do we see 
any significant difference. The gain in mean scores for subscale I (understanding about the 
scientific enterprise) is significant (p < .01) when comparing pretests and posttests (see 
Table 4). When examined using the regression technique that first evaluates the role of 
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time-on-task in explaining variance, we find a significant, albeit weak, main effect 
attributable to internship duration. 

To check the supposition that the 6 week students and 8 week students were already 
significantly different at the pretest level, thus accounting for the differences seen at the 
posttest level, an analysis of covariance was performed. While some differences do exist 
between the prestest scores of the 6 and 8 week groups, once those differences have been 
controlled by the use of the ANCOVA detailed in Table 5, the remaining variance is best 
accounted for by the time on task variable. The difference established by the analysis of 
covariance is significant at the .05 level. 



Please Insert Table 5 Approximately Here 



Discussion and Conclusions 
With respect to the goals of this study, the following conclusions may be drawn. This 
group of high-ability students already possessed substantial knowledge about the nature of 
science before attending the laboratory institute. Subscale III (Understanding the Methods 
and Aims of Science) was the weakest at both the pretest and posttest levels. An 
examination of the individual items in this section reveals that the test authors generally 
targeted scientific methods in the construction of this subset. It may be that these students, 
if they have had typical school science experiences have performed verification rather than 
investigative laboratory activities and have not yet experienced real scientific inquiry. The 
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lack of growth in this domain may be due to the brevity of the summer institute or to their 
limited role in the design of the project on which they worked. Equally interesting, 
however, are the high scores on the other subscales even before the start of the internship. 
There was a fairly high positive correlation between students' SAT verba , scores and the TOUS 
results, so perhaps it is general reading and reasoning ability rather than true knowledge of 
the nature of science that best accounts for the high initial scores. 

McArthur (1993) in her qualitative study of the same interns found that students did 
learn much about the nature of science through their summer experience. However, such 
a claim cannot be so clearly substantiated with the quantitative TOUS study reported here. 
This mismatch between findings -- while supportive of the role of joint quantitative- 
quantitative investigations -- is somewhat troubling. 

The relative lack of growth seen in the TOUS scores reported here may be due to 
the nondiscriminatory character of the instrument itself and the ceiling effect related to the 
high pretest scores or it may be that this instrument is not the ideal tool to use in a study 
of this type. 

Finally, the sole significant result, that the longer one works as a laboratory intern 
the more likely it is that growth in understanding will occur in the domain of knowledge 
regarding the scientific enterprise should be examined. If this is a valid result, it would be 
interesting to discover what aspects of the laboratory internship are responsible for the 
growth seen. Furthermore, if time-on-task is a factor in the increase in knowledge about 
any aspect of the essential character of scientific research, the logical recommendation 
would be to increase the length of time that students spend working in this arena. 
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Implications for Future Research 

Aikenhead (1973), in an analysis of instruments used to measure students' knowledge 
about science and scientists has shown that several assessment tools have been developed 
to provide information about this domain. He indicates, for instance, that the Test on the 
Social Aspects of Science (TSAS) resembles Scale I of the TOUS. Given the indication that 
students participating in laboratc y internships for longer periods do show statistically 
significant growth in Subscale I (understanding about the scientific enterprise), it would be 
useful to apply the TSAS in future studies of the effects of laboratory internships on 
students' knowledge of the nature of science or to establish the correlation between the 
TSAS and subcale I. An analysis on the individual items comprising the subscales to 
suggests explanations for the low discriminatory ability of the test, to provide clues 
explaining the high initial knowledge of the nature of science exhibited by the students or 
to restructure the test making it more effective as a assessment tool. Finally, the generally 
lackluster performance of the TOUS suggests that a new nature of science assessment 
instrument - perhaps designed for students working in laboratory and field environments, 
should be developed. With a renewed emphasis nationally on authentic instruction, it is 
reasonable to assume - and hope - that larger numbers of students will be participating in 
internships as part of their school science experiences. 
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Table 1. Characteristics of Students Attending a Summer Laboratory Internship 



Independent 
Variable 


Mean 


Standard 
Deviation 


Number of 
Students 


Academic Average 
(percent) 


95.79 


4.43 


56 


Grade Completed 


10.77 


0.61 


53 


Number of Science 
Classes Completed 


3.40 


1.06 


57 



Independent Variable 


. Percent of Students Illustrating 
Characteristic 




Male 


47% 


Gender 


Female 


53% 




10 


32% 


Grade 
Completed 


11 


59% 


12 


9% 




Native 
American 


4% 


Ethnicity 


African 
American 


5% 


Asian 
American 


37% 




Caucasian 
American 


46% 




Hispanic 
American 


5% 




No 
Response 


3% 



ERIC 



38 



Table 2. Summary of Studies Investigating the Effect of Inquiry Instruction on 
Students' Knowledge of the Nature of Science. 





Grade 


Type 


Scale I 


Scale II 


Scale 
III 


TOUS 
Total 


Mallette 
(1967) 


Jr High 
Science 


Int. 


n.s. 


n.s. 


sig. 


n/r 


Kendall 
(1970) 


11- 12th 
Physics 


Int. 


n/r 


n/r 


n/r 


sig. 


Mackay 
(1971) 


7-10th 
Science 


Int. 


n.s 


n.s. 


n.s. 


n.s. 


Sheppard 
(1969) 


Jr High 
Science 


Inv. 


sig. 


sig. 


sig. 


sig. 


Green 
(1972) 


Jr High 
Science 


Inv. 


n.s 


sig. 


sig. 


n/r 


Schlenker 
(1970) 


5-8th 
Grades 


Inq. 


sig. 


sig. 


n.s. 


n/r 


I Smith 
1 (1971) 


College 
Biology 


Ext. 


n.s. 


n.s. 


n.s. 


n.s. 


1 Durkee 
I (1974) 


Physics 
Lab Int 


Sum. 


n.s. 


n.s. 


n.s. 


n.s. 



Int. = Integrated vs. Traditional Inv. = Investigative (ESCP) vs. Traditional 
Inq. = Inquiry vs. Traditional Ext. = Extended vs. Traditional Labs 

Sum. = Summer Physics Lab Internship 



n.s. = No Significance sig. = Significance n/r = not reported 



Table 3. TOUS Test Statistics for a Group of High Ability Secondary School Student in a 
Summer Laboratory Internship 



Test 


# of 
Items 


Reliability 
KR-20 


Mean 
Difficulty 


Mean 
Discrimination 


PRE 


PST 


PRE 


PST 


PRE 


PST 


Scale 
I 


18 


0.48 


0.38 


72.94 


76.11 


0.32 


0.29 


Scale 
II 


18 


0.22 


0.57 


75.67 


75.50 


0.27 


0.36 


Scale 
III 


24 


0.53 


0.44 


60.04 


60.12 


0.29 


0.26 


TOUS 
Total 


60 


0.64 


0.66 


68.63 


69.30 


0.21 


0.23 



N = 57 



CO 



Table 4. Pretest and Posttest TOUS (Form W) Scores for a Group of Summer 
Laboratory Intern Students with t-Tests. 







Pretest 


Posttest 










Mean 

(%) 


S.D. 


Mean 

(%) 


S.D. 


t 


P 




6 wks a 


70.50 


13.26 


72.74 


11.25 


-1.125 


n.s. 


Scale 
I 


8 wks b 


78.71 


12.55 


84.53 


8.64 


-2.593 


<.01 




Total c 


72.95 


13.49 


76.26 


11.79 


-2.125 


n.s, 




6 wks* 


75.40 


10.54 


74.90 


14.79 


0.235 


n.s. 


Scale 
II 


8 wks b 


76.35 


11.22 


77.12 


10.94 


-0.301 


n.s. 


Total c 


75.79 


10.43 


75.56 


13.70 


0.137 


n.s. 




6 wks a 


60.00 


12.29 


58.45 


10.93 


0.843 


n.s. 


Scale 
III 


8 wks b 


60.41 


13.75 


64.88 


13.36 


-1.775 


n.s. 


Total c 


60.12 


12.62 


60.37 


11.96 


-0.161 


n.s. 




6 wks a 


67.78 


8.25 


67.73 


8.79 


0.036 


n.s. 


Total 

TOUS 

Test 


8 wks b 


70.77 


10.59 


73.47 


8.62 


-1.873 


n.s. 


Total c 


68.67 


9.02 


69.44 


9.06 


-0.726 





Degrees of Freedom = 39 
Degrees of Freedom = 17 
Degrees of Freedom = 56 



6 wk = Scores of summer internship students participating for 6 weeks 
8 wk = Scores of summer internship students participating for 8 weeks 
Total = Scores of all summer internship students 



ERLC 



Table 5. Analysis of Covariance for Posttest Scores by Time on Task with Pretest Scores 



Source of 
Variation 


Sum of 
Squares 


DF 


Mean 
Square 


F 


Significance 
of F 


Covariates 
Pretest 


668.731 


1 


668.731 


38.842 


0.000 


Main Effects 
Time on 
Task 


69.074 


1 


69.074 


4.012 


0.050 


Explained 


737.804 


2 


368.902 


21.427 


0.000 


Residual 


929.705 


54 


17.217 






Total 


1667.509 


56 


29.777 







c 6 



